Hindsight Network Credit Assignment: Efficient Credit Assignment in Networks of Discrete Stochastic Units

Training neural networks with discrete stochastic variables presents a unique challenge. Backpropagation is not directly applicable, nor are the reparameterization tricks used in continuous variables. To address this challenge, we present Hindsight Network Credit Assignment (HNCA), novel gradient estimation algorithm for of units. HNCA works by assigning credit to each unit based on degree which its output influences immediate children network. We prove that produces unbiased estimates reduced variance compared REINFORCE estimator, while computational cost similar backpropagation. first apply contextual bandit setting optimize reward function unknown agent. In setting, empirically demonstrate significantly outperforms REINFORCE, indicating reduction implied our theoretical analysis significant and impactful. then show how can be extended more general outputs network units, where known version train variational auto-encoder it compares favourably other strong methods. believe ideas underlying help stimulate new ways thinking about efficient assignment compute graphs.